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ABSTRACT 

This paper deals with two fundamental problems that 
are embedded in the evaluation of school practices: (1) attributing 
value to outcome measures, and (2) attributing outcomes to particular 
practices. lays of generating evidence, regarding the value of an 
outcome are discussed, particularly longitudinal researth that 
clarifies the causal relations that exist bet^ween an educational 
outcome and subsequent' events. As far as attributing putcomes to 
practices is concerned, it is recommended that the instructional 
practices under investigation be considered as domains of observable- 
dimensions that can explain variation in student outcomes, rather 
thaa as a set of discrete, unique practices. A model is called for 
that can guide the further development of such dimensions. - 
(Author) 
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I have been asked to considet vith you today some of the 
broad issues in evaluation methodology, particularly as they relate 
to needed research and development in educational psychology. 
Specifically, I want to deal with two ftindamental problems that 
are embedded in the evaluation of school practices: 

1. Attributing value to outcome measures. 

2. Attributing outcome effects to particular school 
V practices. , : ^ 



Attributing Value to Outc -nes 

First, I shall deal with the problem of establishing a valued 
set of outcome measures. In the pas t^ innovators have been rather 
inai ^ulate regarding the value of the outcomes of their new cur- 
ricula or programs. Where clear objectives did exist, evaluators 
did not question vrfiere they came from nor assessed their value. 
The task of evaluation was thought to include only the following steps: 
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review the stated objectives of the educational program, locate or 
develop measures of those objectives, and then determine how well 
the program achieved those objectives. In this approach, the main 
question asked about the outcome measure was how well it assessed 
the objecuves of the program. Clearly we must expand this ap- 
proach by recognizing that value statements regarding school out- 
comes are also subject to empirical investigation. If real progress 
is to be made in evaluating schooL programs, it is important- for 
us to •^demystify^* the value field, since all planned educational 
practice is influenced by the -s^lue attached to the educational ends 
Jthat are sought. 

Dewey's (1939) Theory of Valuation is a most convincing 
guide into value inquiry. His central argument is that facts have 
a bearing on values. Dewey put it this way, ''The notion that valua- 
tions do not exist in empirical fact, and i:hat therefore value con- 
ceptions have to be imported from a source outside experience, is 
one of the most curious beliefs the mind of man has ever enter- 
tained [p. 58]." To assert that value propositions are somehow dif- 
ferent from other kinds of propositions and are not empirically test- 
able is to be at the mercy of special interest groups who commend 
values favorable to themselves as universals. Value disputes are 
then settled by power alone, rather than by rational inquiry. Dewey 



wanted us to see that general propositions regarding value can 
and should be grounded in empirically tested cause-and-effect re- 
lationships. • 

Means -ends relations are cause -effect relations, and they 
form the same type of continuum, with means leading to ends which, 
in turn, are means to subsequent ends in a seemingly endless chain. 
A particular outcome measure will be highly valued if it is perceived 
as a^necessary link in a means-end continuum leading to some end-* 
in- view, which arises from a currently perceived need* The end- 
in-view, according to Dewey, is what is desired to meet that need, 
and is continually revised based on the consequences, of pur exper- 
ience. The end actually attained becomes the means for achieving 
future ends. 

There hayp been serious attempts by psychologists to treat 
value and valuation objectively. Hull, in his 1952 book entitljed 
A Behavior System, devoted an entire chapter to ''Value, Valuation 
and Behavior Theory. He showed how .Dewey Vs general approach 
coiild be the basis for a psychological theory of values and valuing. 
Another example is in Plans and We Structure of Behavior, where 
Miller, Galanter, and Pribram (I960) presented a model of human 
behavior that deals directly with values. They proposed that each 
of us establishes an internal cause-and-effect representation or 



T nibdel of the universe. This cognitive map of how the world works, 
\^ich they called the Iniage, was thought to control our actions. 

- " Mller and his colleagues further proposed that values are part of 
this Image, and are needed in order to cope with the problem of 
choice. They pointed out that values are basediupon the empirical 
knov/ledge that forms the person's Image. 

^ Individuars do operate as if they had a causal model of the 
world that guides their behavior. One contribution of attribution 
theory is to reveal the different kinds of causal models that people 
have . When people have different value s, it is because they have 
V, built up different models. Some individusds, for example^ believe 
that in our present society, schooling is the gr^at certification 
mechanism for better jobs. They also believe that success in school 
depends upon the development of general academic ability. There- 
fore, they value school programs that are shown to positively affect 
the development of academic ability, i^hey are not impressed with 
claims that such abilities are not related to success, as adults, be- 
cause their causal model tells them that such abilities are very 
vreleyantT 

Similarly, some people believe that differences among schools, 
.teachers, educational programs, etc., do make a difference in what 
children learn. Thus, they are not particularly impressed when they 
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are told that school differences do not affect achievement. Few 
parents' 5eem to be willing to run the risk of having their childr 
^ . attend ''poor schools" solely on the basis of one study indicating 

^ that school differences have no effect on the development of academic 
ability. They value good schools because they believe good schools 
lead to better abilities, which lead to better jobs, more meaningful 
-Uves, etc. ^ : v. 

In order to illustrate the kinds of research that l am calling ^ 
for herfe, let us exaimne the evidence regarxfing the v^ : } : 

: inte£Lectual development as a school outcome. General inteUectual - 
development is a summary of the current status of the cognitive de- r 
velopment of the individual student. In a sensie, f it is the current 
profile level for a variety of cognitive tMks that sample diff _ - 

aspects of scholastic ability. It can be mea&ured as the principal ' : 

component of just about any battery^of cognilive tasks currently ' 
available. The exact nature of tlie battery is not important because _ ^ 
of the pervasive nature of this general factor, as shown by the high 
degree to which the principal component from one battery corre- 
lates with that from another. 

The transfer value of general intellectual development within 
schooling is extremely well established. No one seems to doubt that 
/ the general factor, when measured at one educational level> is by 




far the be^t predictor of academic 

Its established within- school predictive validity.is; the main reason 
- why' it is the primary basis for college admis sions decisions . 

Where the value of general intellectual develppnient seems 
to be controversial is inits relationship to tlll^^ V 
mance of both students and individuals who have completed formal 
training. Some studies, : such as those by Holland and Richards 
(1965, 1967) and Wallach and Wing (T969), reveal a certain lack of ^ 
relationship between general intellectual development and selected t^r 
lented accomplishments that occur outside the classroom. Hqyt (1965) 
summarized 46 studies that, on the whole, suggested that college grades 
tend not to be related to earnings within occupatiohs^l groups,; Je — fcs ; 
et al. (1972) reviewed these kinds of studiejs and ext^ the. _ 

generalization that success in schooling^^ is unrelated to success as 
^ an a4xilt.. In a report of some recently cpmpleted re search, Munday 
and Davis (1974) concluded that, "As we evaluate coUege outc^ 
terms of pbstcoUege student behaviors, we^inay have to reappraise - 
the central role previously assigned aca i].*' 

In contrast, the Project TAliEOT research (e. g. , Cooley 
& Lohnes, 1968) showed that general intellectuai development is by_ 
far the most important single predictor of the occupational sorting 
that takes place subsequent to the termination of schooling. Although 



ERIC 



it may not be highly related to criteria such as earnings within an 
occupational group, it quite clearly distinguishes among occupa- 
tional groups. It is even quite highly related to occupational group 
membership for people who have reached the same level of educa- 
tional a tlainmenf. _ ' 

^ These studies illustrate that the relationship between general 

intellectual ability and adult accomplishment is clearly a re searchable 
question, and is very much in need of further clarification. The kind 
of research that is relevant to the value of an outcome measure is lon- 
gitudinal research that clarifies the causal relatiorfa^ that exist between 
the educational outcome and subsequent events. It is ^^^kind of re- 
search that must be \mdertaken if thereHs to be a convincingNt»sts'" 
for the selection of appropriate dependent variables for studying^^e 
^ffects of school programs. If we want others to accept new outconrt 
Measdre^s, their value must b 

It is important to note at this point that a measure's value ^ 
as an indicator of school outcomes must be distinguished from it s^ 
utility for other purposes. For example: 

1. Qeneral intellectual development is not neces- 
sarily useful for making instructional decisions 
* , regarding iiidividiial children. , 
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2. It is not necessarily valid for screening appli* 

cants for particular kinds of jobs. - ^ 

3. Other, more specific measures may be better 
predictors of specific extra-school perfor- 
mances and have better vali^ty. v 

Certainly, other measures are more appropriate than general aca- 
demic talent in many situations, but that fact is not relevant to the 
value of general intellectual development as a general measure of 
school outcomes. Abuses in the application of a measure represent 
a podr argument for never using it. 

In addition to establishing an outcome' sipredictive validity 
in longitudinal studies, there are other ways to generate evidence 
regarding the yalue of an putcozne measure. For example, an analyr 
sis of the kinds of knowledge and abilities needed by adults that are 
not now being provided by schools would suggest outcomes that woiild 
be valued. One could also determine valued outcomes through in- 
quiry into the needs of society, and the current needs of students 
while they are in school, as opposed to needs following graduation. 

In selecting outcome measures, there are at least two other 
criteria that musVbe considered in addition to value. First of all, 
there must be some theoretical or empirical basis for expecting that 
the outcome being measured can be affected by the school program 
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or practices under consideration* If, for exannple, general intellec- 
tml development were purely a function of one's biological equipment, 
then it would not be an appropriate outcome measure for assessing 
the effects of school programs. But since it has been shown to be, 
at least in part, a function of environmental conditions, it can be used 
as an outcome. Secondly, for reasons of interpretive simplicity, put- 
comes must be limited to a small set of relatively independent mea- 
sures. Elsewhere, Lohries and I outline the reasons for this and 



provide an example of such a set derived from the Project TALENT 
battery (Cooley & Loluies, in preparation). It s>ems to be widely 



recognized that a broader spectrum of outcomes-must be considered 



in the evaluation of school practices, but this will not3Happen unless 

such measures are built and their value dsmonstrated. 

_ -- " ___ _ - - -- 

The problem of attributing value to outcomes clearly goes 

beyond what has traditionally been included in evaluation research, 
but it seems to me to be the area in greatest need of clarification 
today. Here I have focused upon just one aspect of the problem, es- 
tablishing the transfer value of outcomes. As we begin to delve more 
deeply into this area, other aspects wiir have to be dealt with, such 
as the establishment of need, which surely must go beyond the kind 
of superficial survey that characterizes needs->assessment today. 
Another concern that must be investigated is the ^^y in which new 
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facts, which have a bearing upon values, become part of one's cause- 
effect model of the world. It is one thing to provide evidence rele- _ 
vant to the caxisal relations between school outcomes and subsequent 
events, but quite smother to have individuals incorporate these rela- ^ ^ 
tions into their causal model of the world, particularly if the rela- 
tions are inconsistent with their own model. " , 

- Attributing Outcomes to Practices 

Given a set of vaiued outcomes, the problem of attributing 
outcome effects to particular school practices remains. In the, pa st; ^ „ ^ 
the most typical approach to this problem has beeii to follow the die- : — \ 

tates of experimental design. Experimental and control groups were 
established, with the experimental group pa rticipsi ting in the new 
school practice, while tlie control group followed the ''traditional** 
one. Whenever possible, randomization was introduced in an attempt 
to ••control" the "imcontrollable.*' Different curricida or different - /' 
school practices were treated as if they were quite different and dis- 
tinct. Thus, if brand A was compared to brand B, the ways in which; 
A and B welre similar or different went unnoticed, as did the ways in 
which schools and teac|iers differed in the manner in which they used 
either A or B. 

One of the most serious difficulties witK this general approach 
is that new school practices established in a variety of c^i^assrooms 
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by a variety of teachers cauinot be represented as a homogeneous 
treatment in a fixed effects design. It is very safe to assume that 
new educational practices will vary in significant ways when imple- 
mented in two or more classrooms. In view of this known variation 
in implementation, a major challenge that faces evaluative research 
is to devise good measures of program implementation, and to de- ^ ^ 
velop an acceptable methodology for sorting out treatnient effects 
when treatments vary in uncontrollable ways. 

The recommendation developed here is that the instructioi« 
practices under investigation in a particular program of evaluative 
: research be considered as a multidimensional domain, rather than 
as a set of discrete, unique practices. That is, different curricula 
and instructional methods should be conceived of as differing in de- 
grees along common dimensions, rather than differing in kind or type 

There have been many important attempts to define observable 
; : dimensions of instructional practices. Research conducted by the 
Stanford Research Institute (SRI) on the Follow Through program 
(e.g., Stallings et al. , 1973) is one current example. Tlieir class- 
room observation scheme clearly reveals the differences among the 
various instructional models participating in Follow Through, 
Leinhardt's (1974) research at the Learning Research and Develop- 
ment Center is defining variables that are relevant to variation in 
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the implementation of a particular instructional model. Her work 
is based on assessments of the key features of that models together 
with guesses as to the kinds of distortions in the model one was likely 
to find in the field that wo;4d affect the model* s ' nesH. 
Rosenshine (1971) provides an excellent summary of variables that 
can be derived from teacher-effectiveness research^ which has been 
going on for decades. This research has yielded an assortment of 
variables ranging from teacher warmth to the businesslike character 
of the classroom, with some beginnings toward consistency among the 
trends. - — 

What is needed now is a theory to guide the further develop- 
ment of instructional dimensions. If evaluative inquiry is so designed, 
it could have very important implications for a theory of instruction 
as well ai5 for educational policy. Of course, the purpose of evalua- 
tive inquiry is to provide policy implications, but it could also have 
theoretical implications if it is guided by and designed in terms of 
available theory, no matter how primitive it might be. Also, eval- 
uative inquiry might be more convincing as a basis for policy if it 
is considered in a theoretical framework. 

A problem, of course, is the fact that attempts to develop 
theories of instruction are a relatively recent phenomenon, and 
theories that are presently available are indeed primitive. Bruner 
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(1966) has proposed that a theory of instruction will be quite different 
from available theories of learning* He suggested that instructional 
theory is a prescriptive theory for the optimization of learning, as 
opposed to a descriptive theory of learning that has occurred* Glaser' 
(1970) model for designing and building an instructional program is an 
excellent example of that prescriptive approach. 

In contrast, Carroll (1963) has proposed a conceptual model 
of the factors that affect success in school Iv^arning* The basic propo- 
sition in Carroll's model is that **the learner will suceedj^a learning 
a given task to the extent that he spends the amount of time that he 
needs to learn the task J* Essentially, his quite simple model pro*- 
posed that the degree of learning that takes place in the course of in- 
struction is a function of the time actually spent in direct learning 
activities divided by the time needed* Further, he proposed that the 
time actually spent in learning would be a function of the opportunity 
provided for learning, the amount of time the learner was willing to 
actively engage in learning, and the amount of time that was needed 
to learn, plus an increment that was a function of the quality of instruc 
tion or the student's lack of ability to understand less than optimal 
instruction. - 

In dealing with the problem of attributing outcome effects to 
particular school practices, la Carroll-type approach will probably 

14 



have greater utility than a prescriptive model. What is sought is a 
model of the instructional domain that will specify the kind and amount 
of learning that will occur in a given educational environment, typi- 
cally a classroom. In a reanalysis of some of the SRI Follow Through 
data, Emrick and I have shown that a modification of the Carroll mod- 
el is very useful in organizing a large number of classroom variables 
into a manageable set of four classroom descriptors which, in turn, 
are very useful explanatory measures of outcome effects (Cooley & 
Emrick, 1974). The four classroom dimensions were based upon 
available SRI variables thought to be related to opportunity to learn> 
degree of curriculum structure, quality of the instructional events, 
and classroom events increasing student motivation. What remains to 
be done is to derive a set of classroom measures from such a concep- 
tual model (as opposed to selecting a set from an available data bank) 
and to further test its validity using a variety of classroom settings. 

The purpose of this discussion of classroom dimensions has 
been to illustrate the kind of model that is essential if we are going 
to move forward our ability to attribute outcome effects to educa- 
tional practices. A convincing theory or conceptual model of the 
phenomena is much more essential to causal inference than is, for 
example, random assignment of subjects to treatments in an or- 
thogonal design. As indicated earlier, in studying school practices 
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the design does not stay orthogonal anyway because of implementa- 
tion variation. Also, laboratory-like controls may introduce arti- 
ficialities that limit generalizability to field situations. This prob* 
lem is far more serious than having certain ambiguities introduced 
through less than perfect experimental control, particularly since 
we can guard against generalizing about uon -orthogonal confounding 
if we directly measure the treatment dimensions. Of course, there 
are those who argue that our knowledge of what is important in class- 
room environments is still too primitive to allow us to construct a 
convincing model. I believe that the development of a model is now 
possible and desirable. It is certainly essential that we reduce the 
hundreds of variables thja-t investigators have considered in their 
attempts to find ways in which classroom/teacher differences make 
a difference in what children learn. Doing so will allow us to make 
significant progress in the evaluation of school programs. 

But lis It Evaluation ? 

I suspect that for many of you I have strayed way beyond a 
consideration of evaluation methodology. What I have done is suggest 
that some rather fandameatal research is required before we will 
significantly improve the evaluation of school practices. I don't mean 
to assign such research to people called ''evaluator, V* nor necessarily 
to include it in a set of activities called evaluation research. Research 
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on the troktisfer value of outcomes and the development of a model 
of classroom learning that can explain variation in student out- 
comes represent a set of concerns that should be very relevant to 
psychologists interested in applying their discipline to critical edu-* 
cational problems. 
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